Cache lightweight agentic workflow logs by Copilot · Pull Request #49082 · github/gh-aw

Copilot · 2026-07-30T06:57:35Z

Scheduled log and audit workflows repeatedly download the same artifacts, causing unnecessary transfers and timeouts. Add a two-day shared cache containing 30 days of activation and usage data.

Cache refresh
- Run daily at 1 AM PST.
- Build gh-aw, prune runs older than 30 days, and download only activation and usage.
- Publish a concise step summary.
Cache consumers
- Restore the cache read-only for scheduled workflows using gh aw logs or gh aw audit.
- Enforce the two-day TTL locally.
- Restore after custom checkout steps to prevent checkout cleanup from deleting cached data.
Artifact-aware downloads
- Persist markers for successfully downloaded artifact sets.
- Fetch only missing requested artifacts, including after artifact directories are flattened.
- Require a complete-download marker before satisfying unfiltered audit requests.
- Skip heavyweight workflow-run logs for activation/usage-only downloads.

Warning

threat detection engine error
The threat detection engine encountered an error and could not complete analysis. This is a tooling failure, not a security finding.

Details

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Run: https://github.com/github/gh-aw/actions/runs/30531795286

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 9.62 AIC · ⊞ 8.7K · ◷
_{Comment /souschef to run again}

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan

@copilot remove TTL enforcement

pelikhan · 2026-07-30T07:04:16Z

+          restore-keys: |
+            agentic-logs-
+
+      - name: Enforce two-day cache TTL


@copilot remove this step

Removed the TTL enforcement step in d09f267.

pelikhan · 2026-07-30T07:04:42Z

+          mkdir -p .github/aw/logs
+          ./gh-aw logs \
+            --start-date -30d \
+            --count 1000 \


@copilot --count 100000

set timeout to 50 minutes

Set --count 100000 and timeout-minutes: 50 in d09f267.

github-actions · 2026-07-30T07:17:37Z

Triage Summary

Category: feature
Risk: medium
Priority Score: 37/100 (impact 22 + urgency 8 + quality 7)
Recommended Action: defer

Adds shared cache for agentic workflow log/audit downloads. Draft, PR is in blocked (merge-conflict) state; useful but not urgent.

Generated by 🔧 PR Triage Agent · auto · 54.9 AIC · ⌖ 5.64 AIC · ⊞ 7.9K · ◷

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-30T07:21:48Z

@copilot remove TTL enforcement

Removed TTL enforcement from the cache refresh and generated consumer restore steps in d09f267.

pelikhan · 2026-07-30T07:52:03Z

@copilot use a single cache id since there is a single producer and many consumers

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-30T08:07:24Z

@copilot use a single cache id since there is a single producer and many consumers

Switched the producer and all consumers to the single agentic-logs cache key in 6e01fde. The producer replaces the immutable prior entry before saving the refresh.

Copilot

Pull request overview

Adds a shared lightweight log cache for scheduled workflows and artifact-aware cache markers.

Changes:

Adds daily cache refresh and automatic cache restoration.
Tracks downloaded artifact sets and avoids heavyweight logs.
Updates consumers to use .github/aw/logs.

Show a summary per file

File	Description
`pkg/workflow/compiler_yaml_runtime_setup.go`	Inserts cache restore steps.
`pkg/workflow/compiler_logs_cache.go`	Detects eligible scheduled workflows.
`pkg/workflow/compiler_logs_cache_test.go`	Tests cache restore generation.
`pkg/cli/logs_run_processor.go`	Validates artifact markers on cache hits.
`pkg/cli/logs_run_processor_test.go`	Tests marker-aware cache behavior.
`pkg/cli/logs_flatten_test.go`	Tests marker preservation during flattening.
`pkg/cli/logs_download.go`	Writes markers and reduces log downloads.
`pkg/cli/logs_artifact_set.go`	Implements artifact marker storage.
`pkg/cli/logs_artifact_set_test.go`	Tests marker matching and validation.
`.github/workflows/safe-output-health.lock.yml`	Restores the shared cache.
`.github/workflows/prompt-clustering-analysis.md`	Uses the shared log directory.
`.github/workflows/prompt-clustering-analysis.lock.yml`	Compiles clustering cache changes.
`.github/workflows/portfolio-analyst.lock.yml`	Restores the shared cache.
`.github/workflows/daily-security-observability.md`	Routes downloads through shared storage.
`.github/workflows/daily-security-observability.lock.yml`	Compiles observability cache changes.
`.github/workflows/daily-safe-output-optimizer.lock.yml`	Restores the shared cache.
`.github/workflows/daily-evals-report.lock.yml`	Restores the shared cache.
`.github/workflows/daily-cli-tools-tester.lock.yml`	Restores the shared cache.
`.github/workflows/daily-ambient-context-optimizer.lock.yml`	Restores the shared cache.
`.github/workflows/agentic-token-optimizer.lock.yml`	Restores the shared cache.
`.github/workflows/agentic-token-audit.lock.yml`	Restores the shared cache.
`.github/workflows/agentic-logs-cache.yml`	Defines daily cache refresh and publication.
`.github/workflows/ab-testing-advisor.lock.yml`	Restores the shared cache.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 23/23 changed files
Comments generated: 3
Review effort level: Medium

 func (c *Compiler) generateRuntimeAndWorkspaceSetupSteps(yaml *strings.Builder, data *WorkflowData, needsCheckout bool) bool {
 	runtimeSetupSteps, customStepsContainCheckout := c.prepareRuntimeSetupAndCheckoutInfo(data)
+	if customStepsContainCheckout {
+		runtimeSetupSteps = append(runtimeSetupSteps, sharedLogsCacheRestoreSteps(data)...)


+		if markers, markerErr := os.ReadDir(filepath.Join(outputDir, downloadedArtifactsMarkerDir)); markerErr == nil {
+			for _, marker := range markers {
+				if !marker.IsDir() {
+					dirs = append(dirs, marker.Name())
+				}


+          path: .github/aw/logs
+          key: agentic-logs


gh-aw-bot · 2026-07-30T08:56:00Z

Warning

threat detection engine error
The threat detection engine encountered an error and could not complete analysis. This is a tooling failure, not a security finding.

Details

The threat detection engine failed to produce results.

Review the workflow run logs for details.

@copilot please run the pr-finisher skill, address the unresolved review feedback below, make sure the ADR requirement is satisfied, and rerun checks once the branch is up to date.

Unresolved feedback to address (newest first):

github-actions review thread on pkg/cli/logs_download.go: Cache lightweight agentic workflow logs #49082 (comment)
github-actions review thread on pkg/cli/logs_download.go: Cache lightweight agentic workflow logs #49082 (comment)
github-actions review thread on pkg/workflow/compiler_logs_cache.go: Cache lightweight agentic workflow logs #49082 (comment)
github-actions review thread on pkg/cli/logs_download.go: Cache lightweight agentic workflow logs #49082 (comment)
github-actions review thread on pkg/cli/logs_artifact_set.go: Cache lightweight agentic workflow logs #49082 (comment)
github-actions review thread on pkg/workflow/compiler_logs_cache.go: Cache lightweight agentic workflow logs #49082 (comment)
github-actions review thread on pkg/cli/logs_artifact_set.go: Cache lightweight agentic workflow logs #49082 (comment)

Blocking review status:

github-actions requested changes: missing //go:build !integration build tag in pkg/workflow/compiler_logs_cache_test.go
github-actions requested ADR link/update for the design decision gate

Failed checks:

None listed in compact candidate data, but requested-changes reviews are still blocking.

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 19.4 AIC · ⊞ 8.7K · ◷
_{Comment /souschef to run again}

Copilot · 2026-07-30T08:56:02Z

@copilot run pr-finisher skill

PR-finisher pass complete (0b4022a).

✅ Reviews — All in-scope threads addressed. Outdated pelikhan threads (TTL removal, --count 100000) were already handled in earlier commits. Six github-actions bot threads addressed in this commit: marker-dir read moved outside loop, complete-marker shortcut, comment-line filtering, retry-path marker documentation, and ArtifactSetAll marker written after non-zip retry. Copilot-reviewer TTL thread intentionally skipped (user explicitly removed TTL enforcement).
✅ Checks (local) — make fmt, make lint, make agent-report-progress all pass. Five pre-existing test failures in pkg/cli (TestAuditCachingBehavior, TestViewWorkflowRun_LocalCache_NoError, TestViewWorkflowRun_WithOwnerRepo_ShowsRunURL, TestViewWorkflowRun_WithSafeOutputs_ShowsSection, TestViewWorkflowRun_EmptyDir_WarnsAndReturnsNil) confirmed to exist before this PR's changes.
❓ Checks (CI) — Stale after agent push; needs human re-trigger.
✅ Mergeable — PR is open, not draft, no conflicts detected.

Actions taken: Moved os.ReadDir(markerDir) outside entries loop; added slices.Contains shortcut so the complete all marker satisfies every filtered request; filtered comment lines in usesSharedLogsCache via strings.SplitSeq; documented intentional marker-write tolerance in retryCriticalArtifacts; extended ArtifactSetAll marker write to the non-zip retry path; fixed pre-existing lock-file drift in daily-cli-tools-tester.lock.yml.

Hand-off: CI must be re-triggered by a maintainer before merge. Pre-existing TestViewWorkflowRun_* failures need separate investigation.

pelikhan · 2026-07-30T09:07:24Z

/matt

github-actions · 2026-07-30T09:08:10Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

github-actions

Skills-Based Review 🧠

Applied /diagnosing-bugs, /tdd, and /codebase-design — requesting changes on two correctness issues.

📋 Key Themes & Highlights

Key Themes

Delete-before-save race (agentic-logs-cache.yml): the old cache is deleted before the new save is confirmed, leaving consumers with no cache if the save step fails.
Inconsistent marker-write error handling (logs_download.go): the primary download path returns a fatal error on marker write failure, while the retry path treats the same condition as non-fatal. Pick one (non-fatal is the right call here).
Text-scan fragility (compiler_logs_cache.go): usesSharedLogsCache replicates an ad-hoc text-scanner pattern rather than reusing the structured step-detection already in the compiler.
Test intent gap (logs_artifact_set_test.go): the prefix-match behaviour exercised in TestFindMissingFilterEntriesUsesDownloadedMarkers is implicit, not documented.

Positive Highlights

✅ Excellent use of a marker-file system to replace the fragile non-empty-directory heuristic — the path-traversal guard in markArtifactDownloaded is a nice safety touch.
✅ shouldDownloadWorkflowRunLogs is a meaningful generalisation of isUsageOnlyArtifactFilter that's easier to reason about.
✅ Strong test coverage across the new ArtifactSet marker logic, including the boundary case where individual markers are present but the all marker is not.
✅ ADR included — good practice for a cross-cutting infrastructure change.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · sonnet46 · 61 AIC · ⌖ 5.28 AIC · ⊞ 7K
_{Comment /matt to run again}

… and add build tag Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot · 2026-07-30T09:23:11Z

[!WARNING]
threat detection engine error
The threat detection engine encountered an error and could not complete analysis. This is a tooling failure, not a security finding....

PR-finisher pass complete (d13fe96).

Blocking issues fixed:

✅ Added //go:build !integration build tag to pkg/workflow/compiler_logs_cache_test.go.
✅ ADR-49082 exists at docs/adr/49082-cache-shared-agentic-logs-for-scheduled-workflows.md (status: Draft). No further update needed to satisfy the design decision gate.

Unresolved review threads addressed:

compiler_yaml_runtime_setup.go (multi-checkout wipe): addCustomStepsWithRuntimeInsertion now accepts a postLastCheckoutSteps parameter. Cache restore is no longer appended to runtimeSetupSteps; instead it is inserted after the LAST root checkout in custom steps. In single-checkout workflows the behaviour is identical to before; in multi-checkout workflows a later root checkout can no longer wipe .github/aw/logs before the gh aw logs/gh aw audit command runs. New test: TestSharedLogsCacheRestoreFollowsLastCheckoutInMultiCheckout.
logs_download.go:945 (case-collision never writes all marker): The marker-write condition is now err == nil || skippedNonZipArtifacts || skippedCaseCollisionArtifacts. After a successful downloadArtifactsByName retry, the all marker is written, preventing an unbounded re-download loop for runs with case-colliding artifact paths.

Checks:

make fmt + make agent-report-progress-no-test (lint + recompile): ✅ pass.
pkg/workflow unit tests: ✅ all pass.
5 pre-existing pkg/cli failures (TestAuditCachingBehavior, TestViewWorkflowRun_*) unchanged — not introduced by this PR.

pelikhan · 2026-07-30T09:23:16Z

Copilot should ensure logs download properly, minimize the number of artifacts that need to be re-downloaded and unzipped, and accurately detect when the required workflow data has already been downloaded. The goal is to reuse any pre-downloaded data and avoid unnecessary API calls. @copilot

…imize API calls Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-07-30T09:44:56Z

Copilot should ensure logs download properly, minimize the number of artifacts that need to be re-downloaded and unzipped, and accurately detect when the required workflow data has already been downlo...

Three concrete improvements committed in the latest push:

Unfiltered all-marker fast-path no longer requires a cached summary – the complete-download marker alone is now sufficient to skip the download. Previously, a run with all artifacts present but no run_summary.json (e.g. first download before any analysis pass) would redundantly re-download everything.
Incremental unfiltered download – when a run directory is non-empty but lacks the all marker (typical after a previous filtered pass), gh aw logs now lists the run's artifacts, checks which are already on disk via the marker directory, and fetches only the missing ones. For example, if activation and usage were already downloaded in a prior filtered pass, a subsequent unfiltered request downloads only agent, detection, and experiment rather than re-transferring the full set.
all marker written after individual-path unfiltered downloads – when the individual download path is used for an unfiltered request (dockerbuild artifacts present, or an incremental top-up), the complete-download marker is written once all expected artifacts are confirmed present. This ensures future unfiltered requests take the fast-path rather than re-triggering a full download cycle.

Two new unit tests cover: the incremental scenario where per-artifact markers (with full API names like abc123-activation) satisfy subsequent full-name checks, and that the all marker satisfies arbitrary full-name artifact lists.

github-actions · 2026-07-30T09:47:49Z

Warning

threat detection engine error
The threat detection engine encountered an error and could not complete analysis. This is a tooling failure, not a security finding.

Details

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Great work on implementing the agentic workflow log caching feature! 🎯

This PR looks solid:

✅ Well-described — The PR body clearly explains the problem (repeated artifact downloads), the solution (two-day cache), and implementation approach across cache refresh, consumption, and artifact-aware downloads.

✅ Focused — Single feature with coherent scope: caching 30 days of activation and usage data with daily refresh.

✅ Comprehensive tests — New test files and extended coverage for artifact sets, run processing, and cache logic (136 lines in compiler_logs_cache_test.go alone).

✅ Follows core team process — Created by the Copilot SWE agent as documented in CONTRIBUTING.md.

The implementation spans workflow automation (.github/workflows/), Go CLI logic, and compiler cache infrastructure — everything is consistent and well-integrated. This is ready for maintainer review. 🚀

Generated by ✅ Contribution Check · auto · 48.3 AIC · ⊞ 8.7K · ◷

gh-aw-bot · 2026-07-30T09:52:17Z

Warning

threat detection engine error
The threat detection engine encountered an error and could not complete analysis. This is a tooling failure, not a security finding.

Details

The threat detection engine failed to produce results.

Review the workflow run logs for details.

@copilot please run the skill, address unresolved review feedback if any, and rerun checks once the branch is up to date.

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 9.62 AIC · ⊞ 8.7K · ◷
_{Comment /souschef to run again}

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

done

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

github-actions · 2026-07-31T14:16:35Z

🎉 This pull request is included in a new release.

Release: v0.84.1

Copilot AI and others added 4 commits July 30, 2026 06:26

Add shared agentic logs cache

5d22df7

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Honor artifact-specific cache lookups

8cc5a73

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Require complete cache marker for audits

ce15477

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Preserve artifact cache markers

2462fe9

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI assigned Copilot and pelikhan Jul 30, 2026

Copilot created this pull request from a session on behalf of pelikhan July 30, 2026 06:58 View session

Copilot AI requested a review from pelikhan July 30, 2026 06:58

Copilot finished work on behalf of pelikhan July 30, 2026 06:58

pelikhan reviewed Jul 30, 2026

View reviewed changes

Copilot started work on behalf of pelikhan July 30, 2026 07:06 View session

github-actions Bot added pr-action:defer pr-agent:github-copilot pr-priority:medium pr-risk:medium pr-type:feature labels Jul 30, 2026

Remove agentic logs cache TTL

d09f267

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 30, 2026 07:22

Copilot AI requested a review from pelikhan July 30, 2026 07:22

Copilot started work on behalf of pelikhan July 30, 2026 07:52 View session

Use single agentic logs cache key

6e01fde

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 30, 2026 08:07

pelikhan marked this pull request as ready for review July 30, 2026 08:10

Copilot AI review requested due to automatic review settings July 30, 2026 08:10

Copilot started reviewing on behalf of pelikhan July 30, 2026 08:11 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

github-actions Bot mentioned this pull request Jul 30, 2026

[pr-sous-chef] PR Sous Chef run report #49011

Closed

Copilot finished work on behalf of pelikhan July 30, 2026 08:56

Copilot started work on behalf of gh-aw-bot July 30, 2026 08:57 View session

github-actions Bot previously requested changes Jul 30, 2026

View reviewed changes

Fix multi-checkout cache restore ordering, case-collision all-marker,…

d13fe96

… and add build tag Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot finished work on behalf of gh-aw-bot July 30, 2026 09:23

Copilot AI requested a review from gh-aw-bot July 30, 2026 09:23

Copilot started work on behalf of pelikhan July 30, 2026 09:24 View session

Improve incremental artifact download: skip already-present data, min…

73d54a0

…imize API calls Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan July 30, 2026 09:45

github-actions Bot mentioned this pull request Jul 30, 2026

[Contribution Check Report] Contribution Check — 2026-07-30 #49021

Closed

Copilot started work on behalf of gh-aw-bot July 30, 2026 09:53 View session

pr-finisher: no new changes - reviewing unresolved threads

f1ecd46

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Apply remaining changes

9c1ba82

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

pelikhan merged commit 9ea5db4 into main Jul 30, 2026

pelikhan deleted the copilot/add-gh-aw-logs-downloader-cache branch July 30, 2026 10:09

Copilot stopped work on behalf of gh-aw-bot due to an error July 30, 2026 10:09
The session was cancelled by the user.

This was referenced Jul 30, 2026

[docs] Update glossary - daily scan #49117

Merged

Smoke Test: MAI-Code-1-Flash - 30577088540 #49195

Closed

Uh oh!

Conversation

Copilot AI commented Jul 30, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelikhan left a comment

Choose a reason for hiding this comment

Uh oh!

pelikhan Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

pelikhan Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 30, 2026

Triage Summary

Uh oh!

Copilot AI commented Jul 30, 2026

Uh oh!

pelikhan commented Jul 30, 2026

Uh oh!

Copilot AI commented Jul 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

gh-aw-bot commented Jul 30, 2026

Uh oh!

Copilot AI commented Jul 30, 2026

Uh oh!

pelikhan commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Key Themes

Positive Highlights

Uh oh!

Copilot AI commented Jul 30, 2026

Uh oh!

pelikhan commented Jul 30, 2026

Uh oh!

Copilot AI commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 30, 2026

Uh oh!

gh-aw-bot commented Jul 30, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Jul 30, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jul 30, 2026 •

edited

Loading